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METHOD AND SYSTEM FOR PROTECTING PUBLICLY 
ACCESSIBLE NETWORK COMPUTER SERVICES FROM 
UNDESIRABLE NETWORK TRAFFIC IN REAL-TIME 

CROSS-REFERENCE TO RELATED APPLICATIONS 

5 This application claims the benefit of the following U. S. provisional 

applications: "Denial of Service Detection and Tracking", filed September 8, 2000 
and having U.S. Serial No. 60/231,480; "Hierarchical Network Profiling" also filed 
September 8, 2000 and having U.S. Serial No. 60/231,481; and "Denial of Service 
Scrubber" also filed September 8, 2000 and having U.S. Serial No. 60/231,479. 

STATEMENT REGARDING FEDERALLY SPONSORED 
RESEARCH OR DEVELOPMENT 

This invention was made with government support under Contract 
No. F30602-99- 1-0527 awarded by DARPA. The government has certain rights to 
the invention. 

BACKGROUND OF THE INVENTION 

1 . Field of the Invention 

This invention relates to methods and systems for protecting publicly 
accessible network computer services from undesirable network traffic in real-time. 

2. Background Art 

20 Given the explosive growth of the Internet and increasing reliance on 

the Web for accessing information and conducting commerce, there is an 
accelerating demand for solutions to security problems as corporations and others 
launch e-commerce strategies and begin migrating mission critical applications to 
the Internet. Security is now a business requirement — the actual loss in revenue 
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combined with intangible costs in reputation and customer confidence are only 
exacerbated by the fierce competition that the Internet environment fosters. 

The Internet security software market consists of applications and 
tools in four submarkets: firewall software; encryption software; antivirus 
5 software; and authentication, authorization and administration software. There are 
also a number of emerging security submarkets such as virtual private networks 
(VPNs), intrusion detection, public key infrastructure and certificate authority 
(PKI/CA), and firewall appliances. 

Network-based, intrusion detection systems are based on passive 
10 packet capture technology at a single point in the network. Such systems do not 
provide any information as to the source of the attack. 

A firewall is a system for keeping a network secure. It can be 
implemented in a single router that filters out unwanted packets, or it may use a 
combination of technologies in routers and hosts. Firewalls are widely used to give 
15 users access to the Internet in a secure fashion as well as to separate a company's 
public Web server from its internal network. They are also used to keep internal 
network segments secure. For example, a research or accounting subnet might be 
vulnerable to snooping from within. 

Following are the types of techniques used individually or in 
20 combination to provide firewall protection. 

Packet Filter . Blocks traffic based on IP address and/or port 
numbers. Also known as a "screening router." 

Proxv Server . Serves as a relay between two networks, breaking the 
connection between the two. Also typically caches Web pages. 
25 Network Address Translation (NAT) . Hides the IP addresses of 

client stations in an internal network by presenting one IP address to the outside 
world. Performs the translation back and forth. 
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Stateful Inspection . Tracks the transaction in an order to verify that 
the destination of an mbound packet matches the source of a previous outbound 
request. Generally can examine mxiltiple layers of the protocol stack, including the 
data, if required, so blocking can be made at any layer or depth. 

5 A denial of service attack is an assault on a network that floods it 

with so many additional service requests that regular traffic is either slowed or 
completely interrupted. Unlike a virus or worm, which can cause severe damage 
to databases, a denial of service attack interrupts service for some period. 

An example includes a client fetching pages from an HTTP server 
10 for the sole purpose of utilizing the server's inbound or outbound bandwidth. 
Another example is a malicious client setting up streaming media cormections for 
the purpose of exhausting a server's connections and bandwidth. 

W U.S. Patent No. 5,606,668 to Shwed discloses a system which 

fll attempts to filter attack traffic that matches predefined configurations. 

Ill 15 U.S. Patent No. 5,828,833 to Bellovin et al. discloses a system 

which allows correct network requests to proceed through the filtermg device. The 
system validates RPC calls and places the authentication information for the call ui 
a fiher table, allowing subsequent packets to pass through the firewall. 

U.S. Patent No. 5,835,726 to Shwed et al. discloses a system which 
20 utilizes filter rules to accept or reject types of network traffic at a set of distributed 
computing devices in a network (a firewall). 

U.S. Patent No. 5,884,025 to Baehr et al. discloses a system for 
packet filtering of data packet at a computer network interface. 

U.S. Patent No. 5,958,052 to Bellovin et al. discloses a system 
25 which possibly modifies a request distribution (in this case DNS request 
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system strips outbound requests of information, thus keeping the original requestor's 
network information private. 

SUMMARY OF THE INVENTION 

An object of the present invention is to provide a method and system 
5 for protecting publicly accessible network computer services from undesirable 
network traffic in real-time. 

In carrying out the above objects and other objects of the present 
invention, a method for protecting publicly accessible network computer services 
firom undesirable network traffic in real-tune is provided. The method includes 
10 receivmg network traffic destined for the services, and analyzing the network traffic 
to identify an undesuable user of the services. The method further includes limiting 
access of the undesirable user to the services to protect the services. 

The undesirable network traffic may include denial of service attacks. 

The network may be the Internet. 

15 One or more user profiles may be generated fi:om the network traffic 

and wherein the step of analyzing may include the step of comparing the one or 
more user profiles with a predetermined profile to determine the undesirable user. 

The step of generating the one or more user profiles may include the 
step of generating request statistics for the user from the network traffic. 

20 The request statistics may include connection statistics and service 

request distributions. 

The network may be the Internet and wherein the step of generating 
request statistics includes the steps of collecting and correlating Border Gateway 
Protocol (BGP) data from the Internet to obtain the service request distributions. 



-4- 



UOM 0206 PUSP 
1920 

The step of correlating may include the step of identifying a 
topologically clustered set of machines in the Internet based on the data and wherein 
the service request distributions are generated from the set of machines. 

In further carrying out the above objects and other objects of the 
5 present mvention, a system for protecting publicly accessible network computer 
services from undesirable network traffic in real-time is provided. The system 
includes an interface for receiving network traffic destined for the services. An 
analysis engine analyzes the network traffic to identify an undesirable user of the 
services. The system further includes a forwarding engine in communication with 
10 the analysis engine for limiting access of the undesirable user to the services to 
protect the services. 

y| The network may be the Internet and wherem the forwarding engine 

=7 collects and correlates Border Gateway Protocol (BGP) data from the Internet to 

pS obtain the service request distributions. 

5^'^ 15 The forwarding engine may identify a topologically clustered set of 

ll\ machines in the Internet based on the data and wherein the service request 

5° J; disfributions are generated from the set of machines. 

The above objects and other objects, feamres, and advantages of the 
present invention are readily apparent from the following detailed description of the 
20 best mode for carrying out the invention when taken in connection with the 
accompanying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIGURE 1 is a schematic view of a denial of service scrubber (DoS 
scrubber) positioned to protect publicly accessible network computer services such 
25 as an Internet service; 

FIGURE 2 is a schematic view of the DoS scrubber architecmre; 
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FIGURE 3a is a schematic view illustrating single link flow 
measurements as a type of flow statistic extraction; 

FIGURE 3b is a schematic view illustrating switching point 
measurements as a type of flow statistic extraction; 

5 FIGURE 4 is a schematic view illustrating a cross-product space for 

a hierarchical network profiler with incoming flows on the left and outgoing flows 
on the right; 

FIGURE 5 is a schematic block diagram flow chart which provides 
an operational description of the hierarchical network profiler; 

10 FIGURE 6 is a schematic view illustrating an example use of a denial 

of service tracker; 

FIGURE 7 is a schematic view of the architecture of a denial of 
service tracker; 

FIGURE 8 is a schematic view of distributed architecture for global 
15 detection and trace back of denial of service attacks; 

FIGURE 9 is a schematic block diagram flow chart illustrating an 
intra-zone denial of service anomaly detector; 

FIGURE 10 is a schematic view illustrating back-tracking a forged 

packet source; 

20 FIGURE 1 1 is a schematic view illustrating a storm tracker which 

backtracks an attack to its source; 

FIGURE 12 is a schematic view illustrating storm breaker blocking 

an attack; 
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FIGURE 13 is a schematic view illustrating attack and anomaly 

detection; 

FIGURE 14 is a schematic view illustrating backtracking a forged 
traffic source; and 

5 FIGURE 15 is a schematic view illustrating blocking DoS traffic. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

In general, the present invention provides a method and system for 
protecting pubhcly accessible network computer services from undesirable network 
m traffic in real-time and is useful within a larger system which combats denial of 

ill 10 service attacks without requiring any changes to the existing Internet routing 

; infrastructore. This larger system, in general, combines network topology 

m information and coarse-grained traffic statistics fi-om routers to detect, backtrack, 

]n and filter distributed attacks on enterprise networks and websites. This larger 

system exploits information from routers without requiring any changes to the 
ui 15 existing Internet routing infrastructure. The larger system as well as the present 

J: 5 invention works with the existing routing infrastructure deployed at Internet service 

providers, application service providers, and enterprise networks. -The method and 
system is enabled by functionality that the major routmg vendors have put into their 
latest products. 

20 The larger system includes a number of complementary components 

as follows: 

StormProfiler . A set of data minmg and network profilmg techniques 
that are used to define "normal" traffic patterns and set dynamic thresholds that are 
continually monitored for early detection and notification. 

25 StormDetector . A new process for real-time monitoring, detection, 

and notification of denial of service attacks and network anomalies. Continuous or 
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periodic sampling is employed for collecting network statistics and extracting 
network topology information from routers. 

StormTracker . A new protocol for correlating anomalous distributed 
events that enables tracking a denial of service attack back to its source. 

5 StormBreaker . A unique solution for protecting and minimizing the 

impact of denial of service attacks on websites and Web hosting services. This 
solution is based on the unique protocol scrubber technology. 

DoS Scrubber 

In general, the method and system of the invention use Internet 
10 routing data in conjunction with passive traffic data to identify application-level 
denial of service attacks. An example includes a client fetching pages from an 
HTTP server for the sole purpose of utilizing the server's inbound or outbound 
bandwidth. Another example is a malicious client setting up streaming media 
connections for the purpose of exhausting a server's connections and bandwidth. 

15 In the invention, a server's request stream — such as a web server's 

web logs — are monitored to build a profile of requests from. a topologically 
clustered set of machines in the Internet. These clusters are identified by their 
administrative domain. These administrative domains are inferred by examining the 
Internet's BGP routing tables from several points in the Internet. By generating the 

20 server's request profiles for sets of clusters in the Internet, malicious hosts that are 
launching application-level denial of service attacks can be detected. While this 
clustering technique has been used in the past for identifying appropriate web caches 
for minunizing web fetch latency, they have not been applied to detecting denial of 
service attacks. 

25 Once these malicious hosts are identified, their requests can be 

filtered either at the server or upstream in the network. 
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A denial of service scrubber (DoS scrubber) is an actively interposed 
network element or system that removes denial of service attacks from legitimate 
network traffic in real-time. The denial of service scrubber removes a new type of 
denial of service traffic from publicly accessible Internet services. 

5 In particular, the DoS scrubber removes denial of service attacks on 

publicly accessible Internet service. Moreover, it uses data mining techniques to 
remove a class of previously unidentifiable denial of service attacks. This new class 
of attacks appear to the service as legithnate service requests; however, these 
requests are generated by a malicious agent with the sole purpose of denying 

10 resources to servicing legitimate requests. 

' Figure 1 shows an example use of the DoS scrubber. It depicts a 

network server providing a publicly accessible service — a public Web server for 
^- ■ example. The DoS scrubber is interposed between the server and the Internet. As 

such, it sees all the traffic that passes between the server and its remote clietits. As 
15 a public server, both legitimate and malicious users gave equal access to its 
resources. However, by analyzing the service request distributions and packet 
statistics, the DoS scrubber can identify malicious users of the service and either 
; filter completely or throttle back their access. 

Protecting Web (Hypertext Transfer Protocol, or HTTP) services is 
20 one specific application of the DoS scrubber. When scrubbing HTTP traffic, the 
DoS scrubber separates legitunate from malicious Web requests. The scrubber 
leverages the fact that HTTP is layered on top of the TCP transport protocol. 
Because TCP sessions cannot be spoofed — that is the source address cannot be 
forged, due to shared random initial sequence numbers — the client-end of service 
25 requests are clearly and uniquely identified. By pairing a client's unique 
identity — its IP source address — with its connection statistics and request 
distribution, a profile can be constructed through data mining. This profile can be 
compared to the normal profile that is obtained through data mining techniques by 
the scrubber during a training session. This training can also be updated on-line as 
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the system runs. Clients with profiles that are flagged as anomalous are then 
candidates for their subsequent requests to be attenuated or completely filtered. 

Figure 2 denotes the denial of service scrubber's high-level 
architecture. It is comprised of two prunary components: the forwarding and the 
analysis engines. The forwarding engine (FE) has two main responsibilities: 
applying filtering and rate limiting to sets of Internet hosts, and generating request 
statistics. The analysis engine (AE) is responsible for the collection and subsequent 
data mining of the forwarding engine's statistics. Upon detection of malicious 
hosts, appropriate actions are fed back from the analysis engine to the forwarding 
engine for filtering or rate limiting the host's requests. 

The DoS scrubber's forwarding engine serves both as an enforcement 
mechanism and statistics generator. When Internet Protocol (IP) packets enter the 
scrubber, they are given to the forwarding engine. Upon receipt, the FE determines 
if the packets belong to an old request, or are part of a new request. If the' request 
is new, a variety of safeguards remove many of the common types of denial of 
service — such as TCP SYN floods. However, the safeguards also include 
checking to see if requesting client has been determined malicious by the analysis 
engine. If so, the request is deah with in a policy configured manner. For 
example, if the service is not overwhelmed, it may allow the request to happen; 
however it can be throttled back usmg a custom rate limiter. When packets arrive 
that are not discarded, statistics are collected that are later sent to the analysis 
engine. Examples of these statistics include: 

- Size: the request and subsequent reply's size, both in bytes and 
packets. 

Request payload: content of the request at the application layer 
(e.g., HTTP GET string). 

Number of fragments: the number of fragments in the request can 

be used to detect some types of malicious use. 

Number of protocol anomalies: the number of errors in the request's 

protocols. 
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The analysis engine uses the stream of request statistics as a feed into 
a data mining system. The system compares the various client request statistics to 
sets of profiles. There are two sets of profiles: canned and trained profiles . The 
canned profiles represent anomalous behavior at the service level. These canned 
5 profiles can be changed through a control interface to match an administrator's 
specifications. The trained profiles are generated by training on the server's 
genuine request statistics. Sophisticated denial of service attacks that were 
previously unidentifiable can be detected by comparing a client's request distribution 
to those of the profiles. 

10 The system differs from firewalls in that it protects publicly 

accessible services from attack. The system recognizes attacks on edge services and 
adapts the forwarding rules to remove them from the network. Statistics and data 
from service requests are sent from the forwarding engine to the analysis engine. 
These data are then analyzed using data mining techniques to find malicious or 

15 anomalous service request patterns. The analysis engine then feeds this information 
back into the forwarding engine to filter or attenuate access to the public service 
from these inappropriate sites. 

Unlike firewalls, the scrubber does not proxy the connections or 
authenticate access to a service; it forwards statistics from a series of client service 
20 requests to be analyzed for attack behavior. 

Hierarchical Network Profiler (HNP) or StormProfiler 

In general, the hierarchical network profiler (HNP) is a new approach 
to network traffic profiling. It aggregates network statistics using a novel cross- 
product of hosts, network and router interfaces to profile network traffic at a 
25 measurement point. 

In particular, the hierarchical network profiler (HNP) represents a 
quanmm leap forward in the area of network traffic profiling. This technology 
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identifies gross bandwidth anomalies automatically at any point in a network's 
routing infrastructure. 

The goal of network profiling is to construct a model of network 
traffic. The approach the HNP takes is to model the network at the granularity of 
5 network flows. A network flow is defined as "a unidirectional sequence of packets 
that are collocated within time that have invariant feature across all the packets." 
These features may include the source and destination addresses, a protocol type, 
and any application layer port information. An example of an Internet flow is a 
sequence of packets that all have the same IP source and destination addresses, IP 
10 protocol value, and UDP or TCP source and destination ports. 

Figures 3 a and 3b show two ways to measure flow statistics in a 
networking environment: at a single networking link of Figiu-e 3a, and at a multi- 
link switching point of Figure 3b. In the single link case, a measurement device 
sits on a single networking link and constructs flow statistics for the underlying 

15 network traffic. Switchpoint statistics generally require measurement support in the 
hardware, such as Cisco System's Netflow technology, or Juniper Network's 
Internet Processor IPs packet sampling technology and Cflowd. This hardware 
support typically provides the standard flow invariants described above in addition 
to information about the incoming and possibly outgoing interfaces. The HNP can 

20 profile the traffic flows gathered in either of these manners. 

The HNP automatically adjusts to its position in the network by 
identifying the typical traffic source and destination pairs for flows that transit the 
measurement point — e.g., router. The diagram in Figure 4 illustrates the 
possibilities for cross-products of mcoming and outgoing endpoints for transit flows. 

25 The most specific endpoint — at the lowest aggregation level — is a host's IP 
address. When hosts are aggregated into network blocks — such as CIDR 
blocks — fewer endpoint statistics are required. These are represented by the 
middle block of endpoints in Figure 4. Finally, the router's interfaces are the 
highest level of aggregation — and the least specific. The HNP adjusts the amount 

30 of aggregation that it keeps on each interface depending on the level of diversity the 
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flow endpoints exhibit along that interface. This diversity is directly proportional 
to the distance from the measurement interface to the endpoints. For example, a 
router close to a set of enterprise hosts will be able to maintain flow statistics about 
each host — a host corresponds to a flow's endpoint when their number will not be 
5 prohibitive. However, in this example, the other endpoint of the flow may be very 
far from this router. Therefore, the HNP may only keep a profile of its 
measurement interface. This example illustrates the general application of the HNP: 
the HNP keeps a profile for the cross-product of the flows that traverse it. In this 
example, it may keep the cross product AxD for flows destined for the Internet from 
10 this set of hosts. 

Figure 5 represents the algorithm for the Hierarchical Network 
Profiler (HNP). At the beginning of the process iteration, the HNP receives 
network flow statistics from the network-forwarding infrastructure. These statistics 
represent summaries of network traffic that the HNP uses to build its profile. After 
15 receiving a set of flow suimnaries (or records), the HNP iterates over each specific 
flow record. It determines if it is interested m the record; that is, a profile is 
maintained for either the flow's source or destination aggregate. If not, the HNP 
updates the source and destination profile with the flow's statistics. 

The HNP then checks to see if the memory and user-defined 
20 requirements continue to be met. If not, the aggregation level for the profiles is 
adjusted so that the requirements are met. When the aggregation level is met, the 
HNP inserts the statistics into the sample profile. The system then checks to see if 
a sampling window has been crossed. When this occurs, the HNP writes the oldest 
profile to persistent storage, and initializes a new profile. If the sampling window 
25 has not been crossed, the new samples are added to the existing profiles. After the 
iteration over the flow statistics has completed, the system then goes back to query 
for further flow statistics, and begins the process over again. 

The HNP takes many available parameters into consideration when 
constructing a traffic profile based on temporal parameters, static network 
30 parameters, and dynamic routing parameters. Temporal parameters are important 
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to discern important differences in traffic behavior. The most important temporal 
properties are: time of day, day of the week, day of the month, and holidays. 
Additionally, the HNP uses static network parameters to gauge the importance of 
downstream hosts and networks for aggregation purposes. Similarly, dynamic 
5 routing information can be used as an input parameter to the HNP. Together 
dynamic routing and topology information form a powerful mechanism for 
identifying salient network flow characteristics. 

The following list describes several applications of the HNP: 



Detecting Denial of Service Attacks : The HNP is very good at 
detecting gross anomalies m network behavior between network 
endpoints. These types of anomalies are the exact signatures left in 
the wake of denial of service attacks. As such, the HNP provides a 
basis for detecting denial of service attacks. 
Traffic Characterization : The HNP can be used for capacity 
planning and traffic characterization. 

Configuration Management : Bugs in network configurations often 
manifest themselves as a change in the network's end-to-end 
behavior. The HNP can easily detect these types of configuration 
problems. 



20 The Hierarchical Network Profile (HNP) differs from past attempts 

to profile network traffic in two ways. First, is uses the network flow statistics 
available both from the routing infrastructure and single link measurement 
infrastructure. Second, it profiles network traffic in proportion to its distance from 
either the source or destination. HNP can profile the network with more accuracy 

25 than traditional approaches by leveraging flow statistics collected directly at the 
router. The second innovation in the HNP is its notion of hierarchy — or distance 
from a packet's source or destination — when constructing a profile. The HNP 
constructs traffic profiles differently, depending on where the measurements are 
collected. Specifically, it keeps track of more information about the flows, the 

30 closer the measurements are collected to the underlying flows' endpoints. This 
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novel approach to profiling allows the HNP to generate useful network profiles at 
any point in the Internet. 

As previously mentioned, StormProfiler represents a quantum leap 
forward in the area of network traffic profiling. This technology allows network 
5 provider and enterprise managers to identify gross bandwidth anomalies 
automatically at any point m their routing infrastructure. Not co incidentally, these 
types of anomalies are the exact signatures left in the wake of denial of service 
attacks. The StormProfiler differs from past attempts to profile network traffic in 
two ways. First, it uses the network flow statistics available from the routing 
10 infrastructure. Second, it profiles network traffic at a router in proportion to its 
distance from either the source or destination. 

StormProfiler can profile the network with more accuracy than 
traditional approaches by leveraging flow statistics collected directly at the router. 
Past profiling attempts have focused on placing passive measurement devices at 

15 points in the network. These only allow for measuring the traffic on a specific link 
between two routers. In contrast, by profiling directly at the routers, StormProfiler 
can determine how specific traffic is typically routed. An analogy would be hiring 
someone to sit by the side of a road and count how many cars are going in one 
direction — this is the old approach to profiling. In the same analogy, the 

20 StormProfiler sits instead at an intersection, and can tell you how many cars from 
each direction went down which fork. Clearly, you know much more about your 
fraffic patterns from studying the behavior at the intersection (the router). In this 
manner, the StormProfiler builds a model over time of how much traffic is routed 
from one point to another at a specific Internet intersection. This profile has several 

25 uses: the foremost for our purpose is denial of service detection. 

The second innovation in the StormProfiler is its notion of 
hierarchy — or distance from a packet's source or destination — when construction 
a profile. The StormProfiler constructs traffic profiles differently, depending on 
where the router is in the network. Specifically, it keeps track of more information 
30 about the flows, the closer the router is to the source (or destination) it is. This 
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novel approach to profiling allows StormProfiler to scale to any point in the 
Internet. 

StormTracker and StormBreaker 

In general, the denial of service detector and tracker is a system that 
5 detects and backtraces Internet denial of service attacks using packet and flow 
statistics gathered directly from the Internet routing and forwarding infrastructure. 

In particular, the denial of service tracker (DoS tracker) is a system 
that detects, backtraces and blocks Internet denial of service attacks. It works by 
gathering packet and flow statistics dkectly from the Internet routmg and forwarding 
10 infrastructure — hereafter called die forwarding infrastructure. By collecting flow 
statistics directly from the forwarding infrastructure, the DoS tracker is able to trace 
DoS attacks that are untraceable by prior art. Specifically, the DoS tracker can 
pinpoint the origin of Internet denial of service attacks that are launched with forged 
source addresses. 

15 The DoS tracker specifically tracks flood-based denial of service 

attacks. These types of attacks attempt to overwhelm either network or end-host 
resources by generating a stream of packets either durectly or indurectly destined for 
a target. Figure 6 shows an example denial of service attack that can be tracked 
through a sample network. The path of the attack traffic goes through Router-A, 

20 Router-B, and Router-C. The most insidious types of attacks hide their origin by 
forging the source Internet Protocol (IP) address on the attack packets. The problem 
this causes for administrators and security officers is that when the target discovers 
itself under attack, it cannot determine its origin; therefore making it impossible to 
shut the attack down. Our key observation is that we can take statistics directly 

25 from the forwarding infrastructure itself to determine the path and origin of the 
attack traffic — even when it is forged. For example, on some types of forwarding 
infrastructure — such as Cisco and Juniper routers — one can interface directly with 
the infrastructure to find out which interfaces are affected by an attack. In the 
example of Figure 6, the inbound and outbound interfaces that the attack travels 
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across can be ascertained. When pairing this information with knowledge of the 
physical and logical topology, it is possible to trace the attack through the network 
to its source. 

Figure 7 illustrates the DoS tracker's overall architecture. It is 
5 comprised of a two-stage hierarchy: collectors and controllers. The collectors 
interface with the forwarding infrastructure; they collect the statistics and report 
those findings to the controllers. The controllers analyze the statistics, looking for 
denial of service attacks and tracking them to their source. 

The DoS tracker's collector takes samples of statistics from the 

10 forwarding infrastructure. The DoS tracker utilizes two types of statistics that 
routers may collect on our behalf: single packet statistics, and flow-based statistics. 
Single packet statistics are those that provide essential information about a set of 
packets entering a forwardmg node — a router. Some of the statistics kept include: 
destination and source IP addresses, incoming interface, protocol, ports, and- length. 

15 After collection, these single packet statistics can be collected from the router for 
analysis. Juniper Network's packet sampling technology is an example of single 
packet statistic support in the infrastructure. Flow-based statistics are statistics that 
describe a set of packets that are related to the same logical traffic flow. The 
concept of flow is generally defined as a stream of packets that all have the same 

20 characteristics: source address, destination address, protocol type, source port, and 
destination port. They may be either unidirectional or bidkectional. Flow statistics 
aggregate a flow's individual packet statistics into a single statistic. Examples 
include a flow's duration, number of packets, mean bytes per packet, etc. Cisco 
System's Netflow and Juniper Network's Cflowd mechanism are widely deployed 

25 flow-based statistic packages. 

Once the controller has received the statistics from the collector, it 
takes one of two approaches to trace the DoS attacks: directed tracing and 
distribution correlation. In directed tracing, one utilizes the knowledge of network 
topology to work backward toward the source of the attack. With distributed 
30 correlation, the controller compares the attack signature with those discovered at 
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Other nodes in the topology. Attacks that correlate strongly are associated together 
and implicitly form the path from the source to the target. Directed tracing relies 
on the fact that one has both the router's incoming interface statistic for an attack 
and the knowledge of the topology to determine what routers are upstream on that 
5 link. With this knowledge, upstream routers can then be queried for their 
participation in transiting the attack. It is usefiil to note that since these upstream 
routers are looking for a specific attack signamre, it is much easier to find the 
statistics of merit. This contrasts with the distributed correlation approach where 
a general attack profile is extracted from every router's statistics to uncover the 
10 global path for the attack. 

After detection and tracmg, the DoS tracker blocks denial of service 
attacks as close to then- source as possible. By taking a global view of the 
Internet — across service providers and network — DoS tracker is able to coordinate 
both the routing infrastructure's ability to filter certain types of traffic in conjunction 

15 with custom filtering hardware that can be incrementally deployed in the network. 
For example, Juniper's Internet Processor II and Cisco's ACL CAR can be utilized 
to download coarse-grained filters that will remove unwanted DoS attacks in real- 
time. Furthermore, the DoS blocker can be used as a way to filter at a fine-grain 
at high speeds in any networking environment, regardless of the routing 

20 infrastructure's unplementation. As a custom hardware solution to blocking DoS 
attacks, the DoS blocker is simply a configurable network filter. The blocker, due 
to its skaplicity of design, is very scalable. 

The DoS Tracker approach differs from conventional network-based 
intrusion detection (NID) in that it uses statistics from the networking infrastructure 

25 itself in contrast to prior art. Prior art in NID systems uses passive measurement 
techniques at a single point in the network to acquire statistics. These point probes 
don't provide any information about the source of a forged attack and are therefore 
useless for tracing denial of service attacks back to their source. Moreover, NID 
systems are single point measurement systems that have very littie support for multi- 

30 node measurement correlation or cooperation and are unable to scale to service 
provider networks. 
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Cisco System's Netflow flow statistics have not been used for 
tracking network attacks. They have only been used for access control and traffic 
billing. Moreover, we have automated a way of polling the Netflow cache in 
contrast to the continuous mode of Netflow operation used by most products. 

5 Juniper's packet sampling technology and Cflowd mechanism have 

not been used for tracing attacks. 

Another novel feature of the present approach is the filtering of denial 
of service attacks upstream in the Internet. Current practice is for a target of an 
attack to stop DoS attacks at their firewall or border router. The present hivention 
10 differs in that it communicates with the networks and routers along the path back 
ft I ■ toward the attacker. When this path is identified, the system can filter the attack as 

close to its source as possible. 

h| As previously mentioned, the distributed approach to global DoS 

JL, attack detection is based on a notion of both hierarchical and neighboring zones. 

15 The philosophy behind this approach lies in the following observation: every 
III detection/traceback node cannot know about all of the outgoing attacks in the 

Internet; instead, these points should only know about the attacks that are occurring 
in their neighborhood. To handle very large scale — Internet wide — DoS detection 
and traceback, the approach utilizes the namral hierarchy of the Internet addressing 
20 scheme. Specifically, the Internet is broken down into manageable portions called 
zones. These zones then communicate with their neighbors, sharing both specific 
and aggregated attack signatures and traceback information. The Internet scales 
because of hierarchy in addressing and routing. Routers and end hosts could not 
route packets if they had to know about all of the endpoints or routes. By 
25 aggregating this information through hierarchy, the Internet is possible. The same 
approach was taken when designing the algorithm for coordinating global denial of 
service detection and traceback. 

Figure 8 provides a graphical overview of how a portion of the 
Internet — consisting, in this example, of three Autonomous Systems 
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(ASes) — could be organized. The figure shows how the size of the autonomous 
system can be accommodated by increasing the corresponding number of zones. 
There are two types of zones: base zones and aggregate zones. A base zone is a 
zone that consists purely of a set of routers. These routers all reside within the same 
5 AS. The local detection and tracing system described above corresponds to the 
detection and tracing system for a base zone. Higher level zones, or aggregate 
zones, can be constructed from sets of base and other aggregate zones. In general, 
a single zone will not span multiple autonomous systems, but this is not strict. 

The zones communicate with each other m a decentralized, 
10 distributed manner using the Anomaly Description Protocol (ADP), similar to the 
way global routing peers communicate using the Border Gateway Protocol. The 
global zone topology is constructed in three ways: local-AS configuration, peer- AS 
configuration, and remote-AS configuration. Zones within an autonomous system 
are configured — a local-AS configuration — to communicate with each other. 
15 Smce they reside within the same administrative entity, their neighbor parameters 
can be set specifically. When crossing autonomous systems between AS peers, 
neighboring zones can also be set according to policy and topology constraints — a 
peer-AS configuration. When connecting zones to a non-ADP enabled AS, a 
resource discovery algorithm is used to determine the closest neighboring zones 
20 through the chain of non-participating peering ASes. 

The zones operate autonomously, and share information about both 
local and remote attacks using the Anomaly Description Protocol. When attacks are 
detected locally, a zone will propagate the attack to its neighbors using the ADP: 
This propagation mcludes the attack's signamre which can be used for both detection 

25 and blocking. When a zone receives an ADP message from one of its neighbors, 
it adds this attack to those the local zone looks for. It is then further propagated to 
other neighboring zones when it is detected locally. ADP messages are therefore 
constrained to their appropriate portion of the Internet, allowing for scalability. 
Moreover, when passing attack information to neighbors, the ADP attempts to 

30 aggregate attack information so that multiple attacks that are described witii the same 
aggregate profile, resulting in a single ADP entry. 
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The StoraiDetector is a mechanism for identifying denial of service 
attacks withm an ISP, a Web hosting service, or an enterprise network. It combines 
a network's dynamic profile — generated by the StormProfiler described 
hereinbelow — with internal static signatures of denial of service attacks to mstantly 
5 identify malicious traffic. This technology utilizes custom algorithms to identify 
denial of service attacks in the reams of incoming traffic flow statistics gathered 
from the routing infrastrucmre. 

Figure 10 demonstrates the utility of the StormDetector system. A 
host in ISP-A is bombardmg a target server in the Web hosting service with a denial 
10 of service attack. However, the attacker is forging the return address on the packets 
in the attack, making is impossible to determine their true origin. The 
StormDetector' s analysis engine receives flow statistics from the routers in the 
[:l target's hosting service. From these statistics, it can detect the attack at some set 

■- of the affected routers along its path. This path leads dhectly from the target to ISP- 

r- 

15 A's border, where the attack originates. This example demonstrates the utility of 
the StormDetector deployed withm a Web hostmg service's network. It can also be 
i!; used in both source and transit networks. 

When employed at an attacker's originating network, StormDetector 
can pinpoint the location of the attacker. In this case, it will backtrack the attack 
20 du-ectly to its source's first-hop router. It may be that the attacker is a zombie 
residing on a compromised machme in an enterprise network. In addition to 
uncovermg those traditional launchpads, StormDetector will be instrumental in 
identifying attacks originating from home machines that connect to the Internet 
through persistent tier-2 ISP's ADSL or cable modem connections. 

25 Figure 9 represents the process for detecting anomalies in the network 

statistics within a single zone. At the start, the system picks a measurement node 
at random. A set of coarse flow statistics or packet header samples is collected. 
This set of statistics is examined for anomalies. These anomalies include both 
clearly defined misuse of the network resources, and also significant differences 

30 between the profile of the various endpoints and the behavior measured in the 
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sample. If any new anomalies are detected in the sample, they are added as 
conditional anomalies, and the collector is updated with these new conditional 
anomalies. Next, a refmed sample is taken with respect to the pending conditional 
anomalies at the collector. The system then looks at the refmed sample of the 
5 network statistics for the presence of both new conditional anomalies as well as old 
anomalies. For each anomaly found, its stams is updated. The system then goes 
through the outstanding anomalies and prunes out any stale ones. Finally, the 
system updates the database with the latest summary statistics for each of the 
outstanding anomalies. The system then repeats, by beginning at the start node. 

10 ' As previously mentioned, StormTracker includes a set of algorithms 

that provide the functionality for tracking anonymous denial of service attacks to 
their sources. These algorithms provide two main functions: directed searching and 
path reconstruction. Directed searchmg is an algorithm for quickly separating the 
attack traffic from the legitimate network traffic — essentially quickly finding 

15 needles in haystacks. By narrowing the scope of the upstream detection- points, 
directed search provides the means for scalable tracking of large-scale attacks. Path 
reconstruction takes multiple measurements of distributed denial of service attacks 
and determines their global topology characteristics. Specifically, given a huge 
distributed denial of service attack, StormTracker allows many statistics collected 

20 from around the Internet to be quickly and robustly correlated to reconstruct the 
attack tree. 

The StormTracker protocol binds these distributed' detection points 
together. This protocol allows multiple autonomous StormDetectors to cooperate 
and exchange attack information, enabling a globally scoped solution. 
25 StormTracker needed a clear definition of denial of service attacks in order to 
communicate effectively. The StormTracker protocol codifies this definition as a 
standard for exchanging attack information between multiple StormDetector 
networks. 
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Figure 11 shows an example of how two systems with 
StormDetectors can cooperate using the StormTracker protocol to trace the attack 
to its origin. 

StormBreaker is another piece of the solution to denial of service 
5 attacks: stopping the attack. Specifically, once StonnDetector and StormTracker 
trace an attack to its origin, the network uses StormBreaker to filter its effects. It 
protects the target by both guaranteeing it full coimectivity to the Internet as well as 
ensuring its ability to provide legitimate clients with service. The StormBreaker 
technology works with both standard network infrastructure and custom filtering 
10 ' technology. Specifically, it can use the filtering abilities of both Cisco and Juniper 
W routers for removal denial of service attacks. In addition to standard networking 

-:: ■ solutions, a custom filtering appliance has been developed that will remove attacks 

i^l from an interposed link at high-speed line rates. This custom solution is based on 

CM the Intel IXP network processor. 

15 The example in Figure 12 shows the use of StormBreaker to block 

!| = a denial of service attack at its source. The attack has comprised a machine in the 

enterprise network and has been attacking a host downstream in ISP-B. Once the 
O attack has been detected and tracked to its origin, StormBreaker determines the 

appropriate filtering response. Specifically, StormBreaker uses knowledge about the 
20 topology and infrastructure components in a network to make the best filtering 
decision. In this example, StormBreaker applies a filtering rule to the attacker's 
router to remove its traffic from the network. 

The overall system solution to denial of service attacks is 
comprehensive, sophisticated, scalable, and effective. The StormTools suite of 
25 solutions detect malicious attacks, as shown in Figure 13, trace them back to their 
origin, as shown in Figure 14, and remove their packets from the Internet, as shown 
in Figure 15. Together they guarantee a host — such as a besieged Web server 
previously left incapacitated and unable to provide service to legitimate 
clients — sustained network coimectivity to legitimate users. 



-23- 



While the best mode for carrying out the invention has been described 
in detail, those familiar with the art to which this invention relates will recognize 
various alternative designs and embodiments for practicing the invention as defined 
by the following claims. 



-24- 



